Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Group commit IdealState updates #13976

Merged
merged 2 commits into from
Sep 14, 2024
Merged

Group commit IdealState updates #13976

merged 2 commits into from
Sep 14, 2024

Conversation

xiangfu0
Copy link
Contributor

@xiangfu0 xiangfu0 commented Sep 11, 2024 •
edited
Loading

We observed IdealState update becomes bottleneck for large table with high throughput segments update.

This PR shamelessly borrowed the idea/logic/code from: https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/GroupCommit.java to group commit IdealState updates.

By running the IdealStateGroupCommitTest test, for my local setup, 2400 updates using 400 threads end up with 8 Zookeeper Writes, this is a significant improvement to relief zookeeper pressure.

@xiangfu0 xiangfu0 changed the title Move IdealState to group commit mode Group commit IdealState Sep 11, 2024
@xiangfu0 xiangfu0 changed the title Group commit IdealState Group commit IdealState updates Sep 11, 2024
KKcorps
KKcorps reviewed Sep 11, 2024
View reviewed changes
pinot-common/src/main/java/org/apache/pinot/common/utils/helix/IdealStateGroupCommit.java
processed.add(ent);
updatedIdealState = ent._updater.apply(idealStateCopy);
// System.out.println("After merging:" + merged);
it.remove();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it safe to modify the queue here while we are iterating on it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, this is processed in single thread

@xiangfu0 xiangfu0 force-pushed the group-commit-helix-ideal-state-update branch 2 times, most recently from e3e6f9a to 0bd3064 Compare September 11, 2024 08:13
@codecov-commenter
Copy link

codecov-commenter commented Sep 11, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 14.28571% with 174 lines in your changes missing coverage. Please review.

Project coverage is 57.90%. Comparing base (59551e4) to head (c63fe61).
Report is 1035 commits behind head on master.

Files with missing lines Patch % Lines
...inot/common/utils/helix/IdealStateGroupCommit.java 7.63% 121 Missing ⚠️
.../core/realtime/PinotLLCRealtimeSegmentManager.java 4.87% 39 Missing ⚠️
...ntroller/helix/core/PinotHelixResourceManager.java 62.50% 7 Missing and 2 partials ⚠️
...g/apache/pinot/common/utils/helix/HelixHelper.java 25.00% 3 Missing ⚠️
...g/apache/pinot/common/metrics/ControllerMeter.java 0.00% 2 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##             master   #13976      +/-   ##
============================================
- Coverage     61.75%   57.90%   -3.85%     
- Complexity      207      219      +12     
============================================
  Files          2436     2613     +177     
  Lines        133233   143270   +10037     
  Branches      20636    21995    +1359     
============================================
+ Hits          82274    82963     +689     
- Misses        44911    53825    +8914     
- Partials       6048     6482     +434
Flag Coverage Δ
custom-integration1 <0.01% <0.00%> (-0.01%) ⬇️
integration <0.01% <0.00%> (-0.01%) ⬇️
integration1 <0.01% <0.00%> (-0.01%) ⬇️
integration2 0.00% <0.00%> (ø)
java-11 57.86% <14.28%> (-3.85%) ⬇️
java-21 57.79% <14.28%> (-3.84%) ⬇️
skip-bytebuffers-false 57.89% <14.28%> (-3.85%) ⬇️
skip-bytebuffers-true 57.74% <14.28%> (+30.01%) ⬆️
temurin 57.90% <14.28%> (-3.85%) ⬇️
unittests 57.90% <14.28%> (-3.85%) ⬇️
unittests1 40.74% <8.02%> (-6.15%) ⬇️
unittests2 27.95% <8.86%> (+0.22%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@xiangfu0 xiangfu0 force-pushed the group-commit-helix-ideal-state-update branch 2 times, most recently from 5a92076 to 1522ef1 Compare September 11, 2024 09:52
@xiangfu0 xiangfu0 force-pushed the group-commit-helix-ideal-state-update branch 2 times, most recently from 47b2e49 to 4d15842 Compare September 11, 2024 16:05
Jackie-Jiang
Jackie-Jiang approved these changes Sep 12, 2024
View reviewed changes
Copy link
Contributor

@Jackie-Jiang Jackie-Jiang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
Can you double check the existing logic and ensure there is nothing rely on only the current update being processed? If there are, we can consider providing 2 fashions, one allow batching and one does not

pinot-common/src/main/java/org/apache/pinot/common/utils/helix/IdealStateGroupCommit.java Show resolved Hide resolved
pinot-common/src/main/java/org/apache/pinot/common/utils/helix/IdealStateGroupCommit.java Outdated Show resolved Hide resolved
@xiangfu0 xiangfu0 force-pushed the group-commit-helix-ideal-state-update branch from 4d15842 to 8e4edc7 Compare September 14, 2024 13:38
@xiangfu0
Copy link
Contributor Author

LGTM. Can you double check the existing logic and ensure there is nothing rely on only the current update being processed? If there are, we can consider providing 2 fashions, one allow batching and one does not

I think all the logics using commit method are moved this this sync best-effort kind of commit logic. I would say this is always better than the single commit.

@xiangfu0 xiangfu0 force-pushed the group-commit-helix-ideal-state-update branch from 8e4edc7 to c63fe61 Compare September 14, 2024 13:49
@xiangfu0 xiangfu0 merged commit 717895b into apache:master Sep 14, 2024
21 checks passed
@xiangfu0 xiangfu0 deleted the group-commit-helix-ideal-state-update branch September 17, 2024 12:30
@Jackie-Jiang Jackie-Jiang mentioned this pull request Sep 24, 2024
Merged
@dinoocch dinoocch mentioned this pull request Oct 12, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Jackie-Jiang Jackie-Jiang Jackie-Jiang approved these changes

@KKcorps KKcorps Awaiting requested review from KKcorps

Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@xiangfu0 @codecov-commenter @KKcorps @Jackie-Jiang